Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM).

BACKGROUND: Evaluating patients' experiences is essential when incorporating the patients' perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. METHODS: We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. RESULTS: The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. CONCLUSIONS: The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.

The digital scribe in clinical practice: a scoping review and research agenda.

The number of clinician burnouts is increasing and has been linked to a high administrative burden. Automatic speech recognition (ASR) and natural language processing (NLP) techniques may address this issue by creating the possibility of automating clinical documentation with a "digital scribe". We reviewed the current status of the digital scribe in development towards clinical practice and present a scope for future research. We performed a literature search of four scientific databases (Medline, Web of Science, ACL, and Arxiv) and requested several companies that offer digital scribes to provide performance data. We included articles that described the use of models on clinical conversational data, either automatically or manually transcribed, to automate clinical documentation. Of 20 included articles, three described ASR models for clinical conversations. The other 17 articles presented models for entity extraction, classification, or summarization of clinical conversations. Two studies examined the system's clinical validity and usability, while the other 18 studies only assessed their model's technical validity on the specific NLP task. One company provided performance data. The most promising models use context-sensitive word embeddings in combination with attention-based neural networks. However, the studies on digital scribes only focus on technical validity, while companies offering digital scribes do not publish information on any of the research phases. Future research should focus on more extensive reporting, iteratively studying technical validity and clinical validity and usability, and investigating the clinical utility of digital scribes.

The Association Between Complications, Incidents, and Patient Experience: Retrospective Linkage of Routine Patient Experience Surveys and Safety Data.

OBJECTIVES: Linkage of safety data to patient experience data may provide information to improve surgical care. This retrospective observational study aimed to assess associations between complications, incidents, patient-reported problems, and overall patient experience. METHODS: Routinely collected data from safety reporting on complications and incidents, as well as patient-reported problems and experience on the Picker Patient Experience Questionnaire 15, covering seven experience dimensions, were linked for 4236 surgical inpatients from an academic center (April 2014-December 2015, 41% response). Associations between complication and/or incident occurrence and patient-reported problems, regarding risk of nonpositive experience (i.e., grade of 1-5 of 10), were studied using multivariable logistic regression. RESULTS: Patient-reported problems were associated with occurrence of complications/incidents among patients with nonpositive experiences (odds ratio [OR] = 2.8, 95% confidence interval [CI] = 1.6-4.9), but not among patients with positive experiences (OR = 1.0, 95% CI = 0.6-1.5). For each experience dimension, presence of patient-reported problems increased risk of nonpositive experience (OR range = 2.7-4.4). Patients with complications or incidents without patient-reported problems were at lower risk of a nonpositive experience than patients with neither complications/incidents nor reported problems (OR = 0.5; 95% CI = 0.3-0.9). Occurrence of complications/incidents only increased risk of nonpositive experience when patients also had problems on "continuity and transition" or "respect for patient preferences" dimensions. CONCLUSIONS: Linking safety data to patient experience data can reveal ways to optimize care. Staff seem able to ensure positive patient experiences despite complications or incidents. Increased attention should be paid to respecting patient preferences, continuity, and transition, particularly when complications or incidents occur.

Patient Experience Monitor (PEM): The Development of New Short-Form Picker Experience Questionnaires for Hospital Patients with a Wide Range of Literacy Levels.

PURPOSE: Several patient-reported experience measures (PREMs) were developed through the years. These questionnaires are frequently found to be inappropriate for people with lower literacy levels. This paper describes the development of patient experience questionnaires for hospital patients with a wide range of literacy levels, while enabling the potential for quality improvement. METHODS: Mixed methods were used to adapt Picker Institute patient experience questionnaires: selection of items and adaptation towards language level B1 (the language level of which patients can express their own opinion and describe experiences, events and expectations) by expert panels, usability tests with patients, analysis of psychometric properties and member checking. A theory-driven approach was followed for definitive enrolment of items, meaning that the items eligible for exclusion had been carefully reviewed by the expert team and representatives of a patient council before definitive exclusion. RESULTS: A pilot study was performed in an University Medical Centre in the Netherlands among in- and outpatients after discharge. Two provisional questionnaires of 22 items, designed by an expert panel, were reduced towards a final selection of 14-15 items. This led to two short-form questionnaires, called Patient Experience Monitor (PEM) Adult Inpatient and PEM Adult Outpatient. To illustrate, the results of the PEM Adult Outpatient questionnaire are presented. CONCLUSION: PEMs are short and valid questionnaires specifically developed to measure patient experiences of hospital patients with a wide range of literacy levels. Acceptance of the questionnaires for both lower and higher educated patients are confirmed by usability tests. The respondents of the pilot study represent both groups. The developed questionnaires should be seen as a dynamic entity and part of a continuous effort to evaluate and improve patient experiences. Future studies are needed to examine the usability of these new questionnaires for quality improvement.

Telemonitoring for Patients With COVID-19: Recommendations for Design and Implementation.

Despite significant efforts, the COVID-19 pandemic has put enormous pressure on health care systems around the world, threatening the quality of patient care. Telemonitoring offers the opportunity to carefully monitor patients with a confirmed or suspected case of COVID-19 from home and allows for the timely identification of worsening symptoms. Additionally, it may decrease the number of hospital visits and admissions, thereby reducing the use of scarce resources, optimizing health care capacity, and minimizing the risk of viral transmission. In this paper, we present a COVID-19 telemonitoring care pathway developed at a tertiary care hospital in the Netherlands, which combined the monitoring of vital parameters with video consultations for adequate clinical assessment. Additionally, we report a series of medical, scientific, organizational, and ethical recommendations that may be used as a guide for the design and implementation of telemonitoring pathways for COVID-19 and other diseases worldwide.

How to automatically turn patient experience free-text responses into actionable insights: a natural language programming (NLP) approach.

BACKGROUND: Patient experience surveys often include free-text responses. Analysis of these responses is time-consuming and often underutilized. This study examined whether Natural Language Processing (NLP) techniques could provide a data-driven, hospital-independent solution to indicate points for quality improvement. METHODS: This retrospective study used routinely collected patient experience data from two hospitals. A data-driven NLP approach was used. Free-text responses were categorized into topics, subtopics (i.e. n-grams) and labelled with a sentiment score. The indicator 'impact', combining sentiment and frequency, was calculated to reveal topics to improve, monitor or celebrate. The topic modelling architecture was tested on data from a second hospital to examine whether the architecture is transferable to another hospital. RESULTS: A total of 38,664 survey responses from the first hospital resulted in 127 topics and 294 n-grams. The indicator 'impact' revealed n-grams to celebrate (15.3%), improve (8.8%), and monitor (16.7%). For hospital 2, a similar percentage of free-text responses could be labelled with a topic and n-grams. Between-hospitals, most topics (69.7%) were similar, but 32.2% of topics for hospital 1 and 29.0% of topics for hospital 2 were unique. CONCLUSIONS: In both hospitals, NLP techniques could be used to categorize patient experience free-text responses into topics, sentiment labels and to define priorities for improvement. The model's architecture was shown to be hospital-specific as it was able to discover new topics for the second hospital. These methods should be considered for future patient experience analyses to make better use of this valuable source of information.

Patient experiences: a systematic review of quality improvement interventions in a hospital setting.

Purpose: In the era of value-based healthcare, one strives for the most optimal outcomes and experiences from the perspective of the patient. So, patient experiences have become a key quality indicator for healthcare. While these are supposed to drive quality improvement (QI), their use and effectiveness for this purpose has been questioned. The aim of this systematic review was to provide insight into QI interventions used in a hospital setting and their effects on improving patient experiences, and possible barriers and promoters for QI work. Methods: Prisma guidelines were used to design this review. International academic literature was searched in Embase, Medline OvidSP, Web of Science, Cochrane Central, PubMed Publisher, Scopus, PsycInfo, and Google Scholar. In total, 3,289 studies were retrieved and independently screened by the first two authors for eligibility and methodological quality. Data was extracted on the study purpose, setting, design, targeted patient experience domains, QI strategies, results of QI, barriers, and promotors for QI. Results: Twenty-one pre-post intervention studies were included for review. The methodological quality of the included studies was assessed using a Critical Appraisal Skills Program (CASP) Tool. QI strategies used were staff education, patient education, audit and feedback, clinician reminders, organizational change, and policy change. Twenty studies reported improvement in patient experience, 14 studies of the 21 included studies reported statistical significance. Most studies (n=17) reported data-related barriers (eg, questionnaire quality), professional, and/or organizational barriers (eg, skepticism among staff), and 14 studies mentioned specific promoters (eg, engaging staff and patients) for QI. Conclusions: Several patient experience domains are targeted for QI using diverse strategies and methodological approaches. Most studies reported at least one improvement and also barriers and promoters that may influence QI work. Future research should address these barriers and promoters in order to enhance methodological quality and improve patient experiences.

Further validation of the Motivation for Traumatic Brain Injury Rehabilitation Questionnaire (MOT-Q) in patients with acquired brain injury.

The Motivation for Traumatic Brain Injury Rehabilitation Questionnaire (MOT-Q) evaluates motivation for rehabilitation in four subscales: Interest in rehabilitation, Lack of anger, Lack of denial, and Reliance on professional help. The objective of this study was to further validate the MOT-Q in 122 inpatients and 92 outpatients with acquired brain injury (ABI). The main measures were motivation for rehabilitation (MOT-Q), self-awareness (Patient Competency Rating Scale), and treatment motivation (Visual Analogue Scale). The MOT-Q showed adequate feasibility in terms of few items with missing responses and few undecided responses. We found no floor or ceiling effects, and significant item-total MOT-Q correlations for 29 of 31 items. Internal consistency was good for the MOT-Q total and acceptable to good for the subscales. The MOT-Q scores were significantly intercorrelated except for the subscales Lack of denial and Reliance on professional help in the inpatient group. The MOT-Q total and subscales were significantly associated with treatment motivation. The Lack of denial subscale showed no significant association with treatment motivation and no to moderate significant associations with self-awareness. In conclusion, the overall MOT-Q is a valid instrument to assess motivation for rehabilitation in patients with ABI. Further research is needed to examine the validity of the subscales.

Dynamic testing of learning potential in adults with cognitive impairments: A systematic review of methodology and predictive value.

Dynamic testing includes procedures that examine the effects of brief training on test performance where pre- to post-training change reflects patients' learning potential. The objective of this systematic review was to provide clinicians and researchers insight into the concept and methodology of dynamic testing and to explore its predictive validity in adult patients with cognitive impairments. The following electronic databases were searched: PubMed, PsychINFO, and Embase/Medline. Of 1141 potentially relevant articles, 24 studies met the inclusion criteria. The mean methodological quality score was 4.6 of 8. Eleven different dynamic tests were used. The majority of studies used dynamic versions of the Wisconsin Card Sorting Test. The training mostly consisted of a combination of performance feedback, reinforcement, expanded instruction, or strategy training. Learning potential was quantified using numerical (post-test score, difference score, gain score, regression residuals) and categorical (groups) indices. In five of six longitudinal studies, learning potential significantly predicted rehabilitation outcome. Three of four studies supported the added value of dynamic testing over conventional testing in predicting rehabilitation outcome. This review provides preliminary support that dynamic tests can provide a valuable addition to conventional tests to assess patients' abilities. Although promising, there was a large variability in methods used for dynamic testing and, therefore, it remains unclear which dynamic testing methods are most appropriate for patients with cognitive impairments. More research is warranted to further evaluate and refine dynamic testing methodology and to further elucidate its predictive validity concerning rehabilitation outcomes relative to other cognitive and functional status indices.

Validity of the dynamic Wisconsin Card Sorting Test for assessing learning potential in brain injury rehabilitation.

The dynamic Wisconsin Card Sorting Test (dWCST) examines the effects of brief training on test performance where pre- to post-test change reflects learning potential. The objective was to examine the validity of the dWCST as a measure of learning potential in patients with acquired brain injury (ABI). A total of 104 patients with ABI completed the dWCST at rehabilitation admission. Performance of a subgroup (n=63) was compared to patients (n=28) who completed a repeated administration of the conventional WCST (rWCST). Furthermore, dWCST performance was compared between patients with ABI (n=63) and healthy controls (n=30) matched on gender, age, and education. Three learning potential indices were used: post-test score, gain score, and a group classification (decliners, poor learners, strong learners, high achievers). The median dWCST administration time was 30 min. The dWCST showed no floor or ceiling effects and the post-test and gain score were significantly intercorrelated. The pre-test score showed no significant associations with other neuropsychological tests. The learning potential indices were significantly associated with language and/or memory. In contrast to the dWCST group, the rWCST group showed no significant pre- to post-test improvement. There were significantly more poor learners in the rWCST group. Compared to controls, patients obtained similar gains, but significantly lower pre- and post-test scores for the dWCST. The ratio of poor learners between-groups was not significantly different. The results support the validity of the dWCST for assessing learning potential in patients with ABI. Further research is needed to investigate the predictive validity of the dWCST.

Impaired self-awareness after acquired brain injury: clinicians' ratings on its assessment and importance for rehabilitation.

BACKGROUND: Impaired self-awareness is a potential obstacle to successful rehabilitation. OBJECTIVE: To obtain clinicians' ratings of the importance of self-awareness for brain injury rehabilitation and use of instruments to assess self-awareness. PARTICIPANTS: One hundred sixty-three members of 3 major Dutch organizations for neuropsychology or rehabilitation. MAIN MEASURE: Online survey addressing: (1) factors participants consider important for the course and success of rehabilitation, (2) whether they assess patients' levels of self-awareness, and (3) the instruments they use to do so. RESULTS: Of the 163 respondents, 116 (71.2%) considered self-awareness to be important for the course of rehabilitation; 113 (69.3%) considered it to be important for the success of rehabilitation. One hundred fifty-six clinicians (95.7%) reported assessing patients' levels of self-awareness, but only 12 (7.4%) reported using standardized instruments specifically designed for this purpose. The instruments most frequently reported to be used were the Awareness Questionnaire and Patient Competency Rating Scale. CONCLUSIONS: It is difficult to capture different aspects of self-awareness in a standardized manner. There is a need for instruments that are valid and reliable and that have good clinical utility.

Validity of the Barrow Neurological Institute (BNI) screen for higher cerebral functions in stroke patients with good functional outcome.

Cognitive impairments are often under diagnosed in stroke patients with good functional outcome. There is a need for a cognitive screening instrument that is sufficiently sensitive to cognitive impairments in these stroke patients. For this goal, we tested the feasibility and validity of the Barrow Neurological Institute Screen for Higher Cerebral Functions (BNIS). Stroke patients with good functional outcome (Barthel Index 19/20) within 1 year poststroke were administered the BNIS and a brief neuropsychological assessment (NPA) including tests for perception, language, memory, attention, reasoning, and executive functioning. We compared the BNIS with the NPA to investigate its feasibility, internal consistency, floor and ceiling effects, concurrent validity, sensitivity and specificity. Fifty-four stroke patients were included. It took significantly less time to administer the BNIS (median = 16 minutes) than the NPA (median = 32.7 minutes). The BNIS showed good internal consistency (alpha = .82) and no floor or ceiling effects. The recommended cutoff values yielded good sensitivity and low to good specificity, depending on age. Except for perception (Spearman correlation .33), BNIS domain scores were significantly (0.44-0.55) associated with matching neuropsychological tests. This study provides promising results for the BNIS as a measure to detect cognitive impairments in stroke patients with good functional outcome.

Development and validation of a short version of the Stroke Specific Quality of Life Scale.

BACKGROUND AND PURPOSE: The Stroke Specific Quality of Life Scale (SS-QoL) is a well validated measure of health related quality of life in patients with stroke, but with 49 items its length is a disadvantage. A short version of the SS-QoL was developed and tested here. METHODS: Secondary analyses of three different studies. The short version was developed using data from 141 patients with aneurysmal subarachnoid haemorrhage (SAH) and tested on data from independent samples of 97 patients with SAH and 105 patients with ischaemic stroke or intracerebral haemorrhage. The item with the highest item domain correlation from each of the SS-QoL domains was selected to obtain a 12 item SS-QoL (SS-QoL-12) with a total score and physical and psychosocial subscores. Criterion validity of the SS-QoL-12 scores was tested in each sample with the original SS-QoL as reference. RESULTS: All three scores of the SS-QoL-12 showed good internal consistency (Cronbach's alpha 0.77-0.89). The SS-Qol-12 scores predicted 88-95% of the variance of the original SS-QoL. Mean differences between the SS-QoL-12 and SS-QoL and their 95% CI were generally within 0.1 points on a 1-5 scale. The limits of agreement were generally within 0.4 points. CONCLUSION: The SS-QoL-12 has good criterion validity for all subsets of stroke. Because it consists of only 12 questions, this short form will be easy to use in research and clinical settings.

Evaluation of change in fatigue, self-efficacy and health-related quality of life, after a group educational intervention programme for persons with neuromuscular diseases or multiple sclerosis: a pilot study.

PURPOSE: An intervention for persons with neuromuscular diseases (NMD) or multiple sclerosis (MS) who experienced severe fatigue was developed which aimed at educating participants in maintaining a balance between capacity and load in their daily activities. This pilot study evaluated the results of this intervention. METHODS: Persons with NMD or MS who experienced severe fatigue were included. Outcome measures were: fatigue (Fatigue Severity Scale), health-related quality of life (HRQoL; SF-36) and self-efficacy (ALCOS-16). Changes in scores between the start of the intervention (T0) and 3 months post-intervention (T1) were tested with the Wilcoxon tests in the complete group and in subgroups (gender, education, high/low self-efficacy). RESULTS: Forty-three persons participated. Significant improvements of HRQoL were found for the domains role-physical, mental health and general health perceptions. Subgroup analyses showed more improvement in males (fatigue, role-physical, vitality, bodily pain, general health perceptions), participants with lower education (role-physical, vitality) and participants with low self-efficacy at T0 (self-efficacy, mental health, general health perceptions) than in females, participants with higher education and participants with higher initial self-efficacy. CONCLUSION: This pilot-study provides preliminary evidence for the effectiveness of a group educational intervention in improving HRQoL without increasing fatigue in persons with NMD and MS.

Validation of the Stroke Specific Quality of Life scale in patients with aneurysmal subarachnoid haemorrhage.

BACKGROUND AND PURPOSE: Disease specific quality of life measures have been validated for patients with ischaemic stroke and intracerebral haemorrhage, but not for patients with aneurysmal subarachnoid haemorrhage (SAH). The aim of this study was to validate the Stroke Specific Quality of Life (SS-QoL) scale for patients with SAH. METHODS: Cross sectional survey of 141 aneurysmal SAH patients. Construct and criterion validity were studied and various ways to merge the 12 SS-QoL domains into a limited number of subtotal scores were explored. Statistics included assessing score distributions, Cronbach's alpha, principal components analysis (PCA) and Spearman correlations between SS-QoL and the Glasgow Outcome Scale (GOS), Cognitive Failures Questionnaire (CFQ), Life Satisfaction-9 (LiSat-9) and Hospital Anxiety and Depression Scale (HADS). RESULTS: PCA revealed two components reflecting physical health and psychosocial health with a mutual correlation of 0.73. A ceiling effect was present for 10 out of 12 domains and for the physical component. Internal consistency was good for all 12 domains (alpha > or =0.80), two components (alpha > or =0.95) and the total score (0.97). Physical SS-QoL scores showed weak to moderate correlations (0.24-0.32) with the GOS. All SS-QoL scores showed moderate to strong correlations (0.35-0.72) with the CFQ, LiSat-9 and HADS. CONCLUSIONS: The SS-QoL is a valid measure to assess quality of life in patients after aneurysmal SAH. Using physical and psychosocial SS-QoL summary scores simplifies the use of this measure without concealing differences in outcomes on different quality of life domains.